Papers with degeneration problem
Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have determined that the learned token embeddings of large-scale neural language models are degenerated to be anisotropic with a narrow-cone shape. |
| Approach: | They propose a method to degenerate the learning gradient for rare token embeddings by gating the specific part of the gradient for all tokens during training stage. |
| Outcome: | The proposed method improves the performance of the models but lacks the training dynamics needed to solve the representation degeneration problem. |
Fine-grained Image Captioning with CLIP Reward (2022.findings-naacl)
Copied to clipboard
| Challenge: | Modern image captioning models are usually trained with text similarity objectives . reference captions often describe only the most salient objects in images . |
| Approach: | They propose to use CLIP to calculate multi-modal similarity and use it as a reward function . they propose a simple finetuning strategy to improve grammar that does not require extra text annotation. |
| Outcome: | The proposed model generates more distinctive captions than the CIDEroptimized model on text-to-image retrieval and fineCapEval. |
A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to address the representation degeneration problem in contextual embedding spaces require a learning process to retrain models with additional objectives. |
| Approach: | They propose a local cluster-based method to address the representation degeneration problem in contextual embedding spaces by removing local dominant directions from verb representations. |
| Outcome: | The proposed method improves CWRs performance on semantic tasks by removing dominant directions of verb representations. |
A Hierarchical Latent Structure for Variational Conversation Modeling (N18-1)
Copied to clipboard
| Challenge: | Variational autoencoders suffer from the notorious degeneration problem, according to a new study . utterance drop regularization is an important feature of the hierarchical RNNs . |
| Approach: | They propose a variational hierarchical conversation RNN framework that exploits latent variables and an utterance drop regularization to exploit latent variable. |
| Outcome: | The proposed model outperforms state-of-the-art models on Cornell Movie Dialog and Ubuntu Dialog Corpus. |
Multi-level Adaptive Contrastive Learning for Knowledge Internalization in Dialogue Generation (2023.emnlp-main)
Copied to clipboard
Chenxu Yang, Zheng Lin, Lanrui Wang, Chong Tian, Liang Pang, Jiangnan Li, Qirong Ho, Yanan Cao, Weiping Wang
| Challenge: | Existing knowledge-grounded dialogue generation models struggle with dull and repetitive outputs, a problem commonly termed as text degeneration. |
| Approach: | They propose a framework that allows the model to "cheat" the objective by duplicating knowledge segments in a superficial pattern matching based on overlap. |
| Outcome: | The proposed framework can be applied to a WoW dataset and shows that it works across models and decoding strategies. |